Main-subject detection method, main-subject detection apparatus, and non-transitory computer readable storage medium

ABSTRACT

A degree-of-saliency map generated from an input image is divided into a plurality of partial areas, and the degree of nonuniformity is calculated from the distribution characteristics of the degrees of saliency of the partial areas. Whether a main subject is present in the input image is judged based on the calculated degree of nonuniformity.

This application is a Continuation of U.S. application Ser. No.15/091,083, filed Apr. 5, 2016, which claims priority from JapanesePatent Application No. 2015-080450, filed Apr. 9, 2015. Theseapplications are hereby incorporated by reference herein in theirentireties.

BACKGROUND Field

Aspects of the present invention generally relate to a technology ofdistinguishing whether a main subject is present in an input image.

Description of the Related Art

Various methods for detecting a main-subject area in an input image areknown. Japanese Patent Laid-Open No. 2000-207564 discloses a method fordetermining a main-subject area based on the location or the size of acandidate object in an input image or based on semantic informationindicating what a candidate area of a candidate object represents.

If such a main-subject area detection process is performed, for example,in an image capturing apparatus such as a digital camera, an autofocus(AF) adjustment and automatic tracking can be performed on a mainsubject.

Some input images, however, do not have a particular subject serving asa main subject. Examples of such images include an image (scenic image)of a distant view that includes the horizon, a skyline, or the like andthat particularly does not have an attention-drawing object. Examplesalso include an image (uniform image) filled with a uniform texture suchas an image filled with a wall or a stone wall. Such images do not havea particular main subject in many cases.

A method for detecting a main subject disclosed in Japanese PatentLaid-Open No. 2000-207564 detects at least one area as a main-subjectarea even in such an image having no particular main subject. Thiscauses the AF, the automatic tracking, and other functions to beperformed on an area that does not correspond to a main subject andcauses a user to experience a feeling of uncertainty.

SUMMARY OF THE INVENTION

An aspect of the present invention provides extracting feature amountsfrom pixels based on at least one region of an input image andgenerating a map in which the extracted feature amounts respectivelycorrespond to the pixels, dividing the generated map into a plurality ofpartial areas and extracting, on a per partial area basis, distributioncharacteristics each based on at least a corresponding one of thefeature amounts, calculating a degree of nonuniformity of the extracteddistribution characteristics of the feature amounts, the degree ofnonuniformity being a degree of spatial nonuniformity, and judging,based on the calculated degree of nonuniformity, whether a main subjectis present in the at least one region of the input image.

Further features of aspects of the present invention will becomeapparent from the following description of exemplary embodiments withreference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block diagram illustrating the configuration of asubject detection apparatus according to a first embodiment.

FIG. 2 is a schematic block diagram illustrating the configuration of amap generator according to the first embodiment.

FIG. 3 is a diagram for explaining a process performed by apartial-image acquisition unit according to the first embodiment.

FIG. 4 is a schematic block diagram illustrating the configuration of adistribution-characteristic extraction unit according to the firstembodiment.

FIG. 5 is a flowchart of a main-subject detection method according tothe first embodiment.

FIG. 6 is a schematic block diagram illustrating the configuration of asubject detection apparatus according to a third embodiment.

FIG. 7 is a diagram for explaining a process performed by apartial-image acquisition unit according to other embodiments.

FIGS. 8A, 8B, and 8C are diagrams schematically illustratingdegree-of-saliency maps generated in the first embodiment.

FIG. 9 is a diagram for explaining patterns of dividing thedegree-of-saliency map in the first embodiment.

DESCRIPTION OF THE EMBODIMENTS First Embodiment

Hereinafter, embodiments of the present invention will be described indetail with reference to the drawings. FIG. 1 is a schematic blockdiagram illustrating the configuration of a main-subject detectionapparatus 101 according to a first embodiment. The main-subjectdetection apparatus 101 includes an image judgment unit 102, amain-subject likelihood calculation unit 107, and a combining unit 108.The image judgment unit 102 includes a degree-of-saliency map generator103, a distribution-characteristic extraction unit 104, adegree-of-nonuniformity calculation unit 105, and a score calculationunit 106.

The main-subject detection apparatus 101 according to the embodiment isimplemented by using a semiconductor integrated circuit (LSI). Themain-subject detection apparatus 101 may also include hardware such as acentral processing unit (CPU), a read-only memory (ROM), a random-accessmemory (RAM), a hard disk drive (HDD), and other components. In thiscase, the CPU runs programs stored in the ROM, a hard disk (HD), andother components, and processes are thereby executed by, for example,components in the functional configuration and in a flowchart (describedlater). The RAM has a storage area functioning as a work area where theCPU loads and runs any of the programs. The ROM has a storage area forstoring the programs and the like run by the CPU. The HD has a storagearea for storing various programs needed for the CPU to executeprocesses and various pieces of data including data regarding athreshold and the like.

Hereinafter, the functional units of the main-subject detectionapparatus 101 will be described. The image judgment unit 102 judgeswhether an input image includes a main subject. Since some input images(hereinafter, also referred to as absent-main-subject images) do nothave a subject serving as a main subject, the image judgment unit 102judges whether an input image is such an absent-main-subject image.Examples of the absent-main-subject image include the scenic image andthe uniform image that are described above. The images do not have asubject serving as a main subject in many cases. Simply performing adetection of a main-subject area on such an image often leads to amisdetection in which a specific area is erroneously detected as a mainsubject.

The image judgment unit 102 according to the embodiment checksdistribution of salient areas in an input image to distinguish whether amain subject is present in the input image. For example, a uniform imagehas almost uniform texture patterns. Even if salient areas attributableto the texture patterns are present, it is expected that the salientareas are uniformly distributed in the entire image. In a scenic image,a horizon or skyline area is likely to be detected as a salient area, itis expected that such salient areas are distributed uniformly in ahorizontal direction. The image judgment unit 102 extracts distributioncharacteristics of salient areas that are specific to such anabsent-main-subject image and thereby judges whether a main-subject areais present in an input image. Hereinafter, a process performed by theimage judgment unit 102 according to the embodiment will be described indetail.

The degree-of-saliency map generator 103 acquires an input image to beprocessed and generates a map of the degrees of saliency(degree-of-saliency map) of the input image. FIG. 2 is a schematic blockdiagram illustrating in detail the configuration of thedegree-of-saliency map generator 103. In FIG. 2, a partial-imageacquisition unit 201 scans the input image using a scan window, cuts outan area image at each extraction location from the input image, andacquires the cut-out area image as a partial image. Data regarding theacquired partial image is output to a distance calculation unit 202.

FIG. 3 is a diagram for explaining a process performed by thepartial-image acquisition unit 201. In FIG. 3, reference numeral 301denotes an input image, reference numeral 302 denotes a first region(inner circular region) of the scan window, and reference numeral 303denotes a second region (outer annular region) of the scan window. Asillustrated in FIG. 3, the partial-image acquisition unit 201 performsan input image scan process on pixel arrays in the input image by usingthe scan window and acquires, as a partial image, an area correspondingto the scan window at each pixel location. In the embodiment, pieces ofimage data of a cut out area corresponding to the first region 302 arereferred to as a first data group, and pieces of image data of a cut outarea corresponding to the second region 303 are referred to as a seconddata group. In the embodiment in this manner, scanning is performed onthe input image by using the scan window, the partial area is cut out ateach pixel location in the input image, and the two data groups that arethe first and second data groups are output to the distance calculationunit 202. Examples of the groups of data regarding the acquired partialimage include low-order feature amounts (such as a luminance value, anedge intensity value, and a texture value) and a combination thereof.

The first region 302 and the second region 303 are respectively definedby the two circular regions (the inner circular region and the outerannular region) in the description above, but the embodiment is notlimited thereto. For example, two rectangular regions (an innerrectangular region and an outer contour region) or regions of anothershape may define the first region 302 and the second region 303. Thefirst region 302 and the second region 303 may also be located in such amanner as to have the centers of the regions (the centers of gravity) atdifferent locations. As long as first and second regions are set asdifferent regions, any of various shapes may be used for the first andsecond regions.

Referring back to FIG. 2, the distance calculation unit 202 calculates adistance between the two data groups that have been input. Since thepartial-image acquisition unit 201 outputs the data groups acquired foreach pixel in the input image every time the data groups are acquired,the distance calculation unit 202 calculates a distance (distance value)between the data groups every time the data groups are input. Tocalculate the distances each between the data groups, any of the variouspublicly known methods are usable. For example, such a method that useshistograms respectively generated for the two input data groups may beused. In the method, an absolute difference between the histograms isobtained for each bin, and the sum of the absolute differences iscalculated. Information regarding each distance value between the datagroups that is a calculation result is output to a map generator 203.

The map generator 203 generates a degree-of-saliency map by using thedistance value calculated by the distance calculation unit 202. Sincethe distance calculation unit 202 calculates the distance valuecorresponding to the pixel location in the input image, the mapgenerator 203 allocates the distance value to the corresponding locationin the degree-of-saliency map. In the embodiment, the distance value isregarded as the degree of saliency, and data taking the form of an image(degree-of-saliency map) is generated.

FIGS. 8A, 8B, and 8C are diagrams schematically illustratingdegree-of-saliency maps generated by the map generator 203 of theembodiment and binary images generated as a result of a binarizationprocess (described later) performed on the degree-of-saliency maps. FIG.8A illustrates an example in which a degree-of-saliency map is generatedfrom an image having a circular object serving as a main subject. FIGS.8B and 8C illustrate examples in which degree-of-saliency maps arerespectively generated from a uniform image and a scenic image that areabsent-main-subject images. In FIGS. 8A to 8C, reference numerals 801,802, and 803 denote input images in the respective examples, andreference numerals 804, 805, and 806 denote degree-of-saliency maps inthe respective examples. The image having a main subject in FIG. 8Aresults in the degree-of-saliency map 804 in which contour lines definesalient areas and the central salient area exhibits a high degree ofsaliency. The uniform image in FIG. 8B results in the degree-of-saliencymap 805 in which a region exhibiting a high degree of saliencyperiodically appears in accordance with a periodic texture of the inputimage 802. In the scenic image in FIG. 8C, the input image 803 has askyline portion that is salient and the color and luminance of which aredifferent from those near the skyline portion. Accordingly, thedegree-of-saliency map 806 has contour lines that follow the skyline.Binary images 807, 808, and 809 will be described later. A processrelated to the degree-of-saliency maps generated by thedegree-of-saliency map generator 103 has heretofore been described.

Referring back to FIG. 1, the distribution-characteristic extractionunit 104 extracts a characteristic indicating the distribution of thedegrees of saliency in the input image by using the degree-of-saliencymap generated by the degree-of-saliency map generator 103. To extractthe distribution characteristic, how pixels exhibiting a high degree ofsaliency are distributed may be checked. In the embodiment, thedistribution characteristic of pixels that exhibit a high degree ofsaliency and satisfy a predetermined condition is checked. Specifically,the degree of saliency of each pixel is compared with a thresholdcalculated based on the degree-of-saliency map. Pixels exhibiting adegree of saliency higher than the threshold are regarded as pixelssatisfying the predetermined condition, and the distributioncharacteristic of such pixels is extracted. In another example, pixelsexhibiting the degrees of saliency that are ranked in a toppredetermined percent in the degree-of-saliency map may be handled aspixels exhibiting a high degree of saliency and satisfying thepredetermined condition.

In the following description, pixels exhibiting a degree of saliencyequal to or higher than the threshold are referred to as white pixels(corresponding to pixels taking on a value of “1” in binarization basedon the threshold), and pixels exhibiting a degree of saliency lower thanthe threshold are referred to as black pixels (corresponding to a pixeltaking on a value of “0” in the binarization). Thedistribution-characteristic extraction unit 104 of the embodiment checksthe distribution characteristic of the white pixels. When checking howthe pixels exhibiting the high degree of saliency are distributed, thedistribution-characteristic extraction unit 104 of the embodimentdivides the degree-of-saliency map into partial areas and extracts thedistribution characteristic of the pixels exhibiting the high degree ofsaliency for each partial area. The distribution-characteristicextraction unit 104 herein checks the distribution characteristic of thewhite pixels but may check the distribution characteristic of the blackpixels.

The details of a process performed by the distribution-characteristicextraction unit 104 will be described by using the schematic diagram ofthe blocks of the distribution-characteristic extraction unit 104 inFIG. 4. In FIG. 4, an extraction condition calculation unit 401calculates the threshold by performing statistical processing on thedegree-of-saliency map, and outputs the threshold as a pixel extractioncondition to a pixel identification unit 403. In the statisticalprocessing for calculating the threshold in the embodiment, a valueranked at a predetermined percentile point of values of the degree ofsaliency in the degree-of-saliency map is set as the threshold.Accordingly, when the threshold thus calculated is used for thebinarization of the degree-of-saliency map, the white pixels account fora percent equal to the top predetermined percent.

A partial-area division unit 402 divides the received degree-of-saliencymap into partial areas. The image judgment unit 102 judges whetherdistribution characteristics extracted for the respective partial areasobtained as a result of division performed by the partial-area divisionunit 402 are similar to or different from each other and performs animage judgment in which whether a main subject is likely to be present.Accordingly, the partial-area division unit 402 desirably divides theinput image into areas appropriately to fulfill the purpose of thejudgment. A method for dividing an input image into partial areas in theembodiment will be specifically described later. The partial-areadivision unit 402 outputs the divided degree-of-saliency map to thepixel identification unit 403.

The pixel identification unit 403 identifies, for each partial area,pixels (white pixels) satisfying the predetermined condition by usingthe pixel extraction condition (threshold) and the degree-of-saliencymap, the pixel extraction condition (threshold) being calculated by theextraction condition calculation unit 401, the degree-of-saliency maphaving partial areas as the result of the division performed by thepartial-area division unit 402. The pixel identification unit 403subsequently extracts the distribution characteristic of the pixels. Inthe embodiment, the pixel identification unit 403 calculates, as thedistribution characteristic, the percentage of the number of whitepixels per partial area. The distribution characteristic (percentage ofthe number of white pixels per partial area) thus calculated is outputto the degree-of-nonuniformity calculation unit 105.

Referring back to FIG. 1, the degree-of-nonuniformity calculation unit105 calculates, as the degree of nonuniformity of the degrees ofsaliency in the input image, the degree of spatial nonuniformity of thedistribution characteristics extracted by thedistribution-characteristic extraction unit 104 for the respectivepartial areas. The spatial nonuniformity herein means how far thedistribution characteristics extracted for the respective partial areasare different from each other. Specifically, the partial areas havingdifferent distribution characteristics have a high degree ofnonuniformity, while the partial areas having the same distributioncharacteristic have a low degree of nonuniformity.

In the embodiment, the percentage of the number of white pixels iscalculated as the distribution characteristic of each partial area.Examples of a method for calculating the degree of nonuniformity in thiscase include a method by which the standard deviation of the percentagesof the number of white pixels is calculated. In the method, as adifference in white pixel percentage between the partial areas isdecreased, the standard deviation is decreased, and the degree ofnonuniformity is also decreased. Contrarily, as the difference in whitepixel percentage between the partial areas is increased, the standarddeviation is increased, and the degree of nonuniformity is alsoincreased. The value used to calculate the degree of nonuniformity isnot limited to the standard deviation. Another statistical valueindicating that the partial areas have different distributioncharacteristics or have the same distribution characteristics may alsobe used for the calculation.

A relationship between how the degree-of-saliency map is divided intopartial areas and the degree of nonuniformity will be described withreference to the degree-of-saliency maps and the binary imagesschematically illustrated in FIGS. 8A to 8C. The embodiment is providedto distinguish between a “present-main-subject image” in FIG. 8A and anabsent-main-subject image such as the uniform image and the scenic imagein FIGS. 8B and 8C. The degrees of saliency calculated for these imagesare illustrated in the degree-of-saliency maps 804, 805, and 806. Inother words, an image having a main subject is expected to produce thedegree-of-saliency map 804 in which areas exhibiting a high degree ofsaliency are distributed locally in a main-subject area. In contrast,the uniform image that is an absent-main-subject image is expected toproduce the degree-of-saliency map 805 in which the salient areasattributable to the texture are uniformly distributed over the image.The scenic image that is an absent-main-subject image is expected toproduce the degree-of-saliency map 806 in which the salient areas aredistributed in the horizontal direction because a horizon or skylinearea tends to be detected as a salient area. In other words, the salientareas in the absent-main-subject image tend to be uniformly distributed,while the salient areas in the present-main-subject image tend to bedistributed locally and nonuniformly.

In FIGS. 8A to 8C, the images 807 to 809 are binary images schematicallyillustrating results of binarization performed on the degree-of-saliencymaps 804 to 806 by using thresholds, respectively. The embodiment isprovided to enable distinction of a present-main-subject image based onthe binary image 807 and an absent-main-subject image based on thebinary images 808 and 809. More specifically, in the embodiment, adegree-of-saliency map is divided into a plurality of partial areas, thedegree of nonuniformity is calculated based on the percentage of thewhite pixels (the distribution characteristic of the degrees ofsaliency) in each partial area, and whether a main subject is present inthe input image is thereby judged based on the degree of nonuniformity.At this time, how the degree-of-saliency map is divided influences thecalculated degree of nonuniformity and is thus important.

A relationship between the degree-of-saliency map division method andthe spatial nonuniformity of the degree of saliency calculable using thedegree-of-saliency map division method will be described with referenceto FIG. 9. FIG. 9 illustrates examples of division patterns used fordividing a degree-of-saliency map into a plurality of partial areas.FIG. 9 illustrates a pattern 901 for dividing a degree-of-saliency mapinto partial areas lengthways and widthways, a pattern 902 for dividinga degree-of-saliency map into partial areas lengthways, and a pattern903 for dividing a degree-of-saliency map into partial areas widthways.For example, when a degree-of-saliency map is divided into partial areaslengthways and widthways as in the pattern 901, two-dimensionalnonuniformity of the degrees of saliency in the input image can becalculated. When a degree-of-saliency map is divided into partial areaslengthways as in the pattern 902, one-dimensional horizontalnonuniformity of the degrees of saliency in the input image can becalculated. When a degree-of-saliency map is divided into partial areaswidthways as in the pattern 903, one-dimensional vertical nonuniformityof the degrees of saliency in the input image can be calculated.

Consider cases where each of the binary images 807 to 809 of therespective degree-of-saliency maps in FIGS. 8A to 8C is divided as inthe patterns 901 to 903 in FIG. 9. For example, if the binary images 807to 809 are divided as in the pattern 901, the binary images 807 and 809exhibit the white pixel percentages that largely differ between thepartial areas and exhibit a high nonuniformity. In contrast, the binaryimage 808 exhibits the white pixel percentages that are approximatelythe same between the partial areas and exhibits a low nonuniformity.Table below lists nonuniformity degree relationships between the binaryimage type and the division pattern.

TABLE 901 902 903 807 (Present-main-subject image) Nonuniformity: HighNonuniformity: High Nonuniformity: High 808 (Uniform image)Nonuniformity: Low Nonuniformity: Low Nonuniformity: Low 809 (Scenicimage) Nonuniformity: High Nonuniformity: Low Nonuniformity: High

The embodiment is provided to judge that a uniform image and a scenicimage are absent-main-subject images, and area segmentation as in thepattern 902 is thus usable to discriminate between anabsent-main-subject image and a present-main-subject image based on thedegree of nonuniformity because the salient areas in each of the uniformimage and the scenic image are uniformly distributed in the horizontaldirection.

In the embodiment as described above, the degree-of-saliency map isdivided into the partial areas lengthways (with partial areas arrangedin a direction corresponding to the horizontal direction of the image)by using the pattern 902. The distribution characteristic of the salientareas specific to the absent-main-subject image is thereby detected, andthe image can be judged. In the embodiment, thedistribution-characteristic extraction unit 104 divides thedegree-of-saliency map into the partial areas lengthways and extractsthe distribution characteristic (white pixel percentage) for eachpartial area. The degree-of-nonuniformity calculation unit 105calculates as the standard deviation of the percentages of the whitepixels as the degree of nonuniformity of the distribution characteristicof the degrees of saliency of each partial area thus set.

Referring back to FIG. 1, the score calculation unit 106 calculates animage judgment score based on the degree of nonuniformity calculated bythe degree-of-nonuniformity calculation unit 105. The image judgmentscore indicates the likelihood of the input image being anabsent-main-subject image. Accordingly, it can be said that the scorecalculation unit 106 functions as a generator that generates theinformation indicating how much the input image is likely to be anabsent-main-subject image. Any type of score may be used as an imagejudgment score, as long as a high degree of nonuniformity of the degreeof saliency produces a low image judgment score. For example, thereciprocal number of the degree of nonuniformity is usable as the imagejudgment score.

In addition, an existing machine learning method may be used as a methodfor calculating an image judgment score from the degree ofnonuniformity. Specifically, if a general class separation method isused for a large number of absent-main-subject and present-main-subjectimages, whether the input image is an absent-main-subject image or apresent-main-subject image can be distinguished based on a specificdegree of nonuniformity. When this method is used, for example, a binaryvalue of 1 as the image judgment score may be output in accordance withdistinction as an absent-main-subject image, and a binary value of 0 inaccordance with distinction may be output as a present-main-subjectimage. In addition, since the likelihood is calculated beforeclassification into classes in many general class separation methods,the likelihood may be output as the image judgment score. When thelikelihood is used, the image judgment score takes on a continuousvalue.

The main-subject likelihood calculation unit 107 calculates main-subjectlikelihood at each pixel location (or in each area) of the image basedon the input degree-of-saliency map. The main-subject likelihoodrepresents the likelihood of the presence of a main subject at eachpixel location (or in each area). The likelihood is herein calculatedbased on a rule of thumb in which a main subject is present in thecenter of the image and is a salient area in many cases. In theembodiment, the likelihood at each pixel location is calculated based onthe degree of saliency at the pixel location and a distance between thepixel location and the center of the image. The main-subject likelihoodmay be set to be decreased as the distance from the center of the imageis increased. In the embodiment, a value obtained by dividing the degreeof saliency at each pixel location by a distance between the pixellocation and the center of the image is used as the main-subjectlikelihood. The main-subject likelihood applicable to the embodiment isnot limited thereto and may be a value calculated by another method.

The main-subject likelihood calculation unit 107 calculates themain-subject likelihood at each pixel location (or in each area) in thismanner. The main-subject likelihood calculation unit 107 subsequentlyoutputs, as a likelihood map, the main-subject likelihood calculated ateach pixel location (or in each area) to the combining unit 108.

The combining unit 108 combines the image judgment score calculated bythe score calculation unit 106 with the likelihood map calculated by themain-subject likelihood calculation unit 107 and outputs the finalmain-subject detection result. As described above, the image judgmentscore calculated by the score calculation unit 106 takes on a binaryvalue or a continuous value.

In a case where a binary value is used as the image judgment score, thevalue indicates whether the current input image is anabsent-main-subject image or a present-main-subject image. Accordingly,if the image judgment score is “1” and thus indicates anabsent-main-subject image, the combining unit 108 judges the input imageas an absent-main-subject image and outputs information indicating thecontent to that effect as a main-subject detection result. If the imagejudgment score is “0” and thus indicates a present-main-subject image, amain-subject detection result may be output based on the likelihood map.Since the main subject is considered to be present at the location wherea high likelihood is exhibited, a score of the likelihood at thelocation is output.

In a case where such a continuous value that is increased when the inputimage is an absent-main-subject image is used as the image judgmentscore, a main-subject detection result may be output based on a map(referred to as the final map) obtained by dividing the likelihood mapby the image judgment score. In this case, if the image judgment scoreis high (if it is judged that the input image is likely to be anabsent-main-subject image), the final map has low values as a whole. Ifthe image judgment score is low (if it is judged that the input image islikely to be a present-main-subject image), the final map has highvalues as a whole. The final map subsequently undergoes predeterminedthreshold-based processing. If there is no area exhibiting a map valueexceeding a threshold, the input image can be judged to be anabsent-main-subject image. If there is an area exhibiting a map valueexceeding a threshold, the area can be judged to be a main-subject area.

Information regarding the main-subject area detected in this manner orinformation indicating the absent-main-subject image is output to anapparatus that performs processing on the main-subject area. Forexample, in an image capturing apparatus such as a digital camera, theinformation regarding the detected main-subject area is used for an AFprocess and an automatic tracking process. In a case where the inputimage is an absent-main-subject image, a main-subject detectionapparatus outputs information indicating the content to that effect.This reduces the forced execution of the AF automatic tracking processon an area that causes the user to experience a feeling of uncertainty.Note that a semiconductor integrated circuit provided in an imagecapturing apparatus such as a digital camera may function as themain-subject detection apparatus. In this case, the image capturingapparatus itself corresponds to the main-subject detection apparatus ofthe embodiment.

Subsequently, a process performed by the main-subject detectionapparatus 101 in the main-subject detection method will be described.FIG. 5 is a flowchart illustrating the procedures of the main-subjectdetection method according to the first embodiment. In step S501, theimage judgment unit 102 checks whether an input image is input in themain-subject detection apparatus 101. If input is confirmed, the processproceeds to step S502.

In step S502, the degree-of-saliency map generator 103 makescalculations and generates a degree-of-saliency map of the input image.In step S503, the main-subject likelihood calculation unit 107calculates main-subject likelihood based on the degree-of-saliency map.In step S504, the distribution-characteristic extraction unit 104subsequently binarizes the degree-of-saliency map generated in stepS502. In the embodiment, the distribution-characteristic extraction unit104 calculates a threshold from the degree-of-saliency map and binarizesthe degree-of-saliency map by using the threshold as a reference.

In step S505, the distribution-characteristic extraction unit 104divides the binarized degree-of-saliency map. In the embodiment asdescribed above, the degree-of-saliency map is divided into a pluralityof partial areas lengthways as in the pattern 902. In step S506, thedistribution-characteristic extraction unit 104 calculates, for eachpartial area resulting from the division, the percentage of the numberof white pixels as the distribution characteristic of each partial area.

In step S507, the degree-of-nonuniformity calculation unit 105calculates the degree of nonuniformity from the percentages of thenumbers of white pixels in the partial areas. The standard deviation ofthe percentages of the numbers of white pixels is used as the degree ofnonuniformity. In step S508, the score calculation unit 106 calculatesan image judgment score from the calculated degree of nonuniformity. Theimage judgment score indicates the likelihood of an input image being anabsent-main-subject image. In the embodiment, the reciprocal number ofthe degree of nonuniformity is used as the image judgment score.

In step S509, the combining unit 108 calculates the final main-subjectdetection result. If it is judged that a main subject is present in theinput image, the final main-subject detection result is output togetherwith the location of the detected main subject and a score (thereliability of the main-subject detection result). If it is judged thata main subject is not present in the input image, information indicatingthe content to that effect is output.

In the embodiment as described above, the degree-of-saliency mapgenerated from the input image is divided into the partial areas, andthe degree of nonuniformity is calculated from the distributioncharacteristic of the degrees of saliency of each partial area. In theembodiment, whether the main subject is present in the input image canbe distinguished based on the degree of nonuniformity.

Second Embodiment

A second embodiment of the invention will be described. The componentsdescribed in the first embodiment are denoted by the same referencenumerals, and description thereof is omitted.

In the second embodiment, the distribution characteristic extractionprocess performed by the distribution-characteristic extraction unit 104and the degree of nonuniformity calculation process performed by thedegree-of-nonuniformity calculation unit 105 are different from those inthe first embodiment. Hereinafter, the differences from the firstembodiment will be described.

First, as in the first embodiment, the extraction condition calculationunit 401 of the distribution-characteristic extraction unit 104 of theembodiment extracts target pixels (white pixels) based on the sizerelationship with the calculated threshold. In the embodiment, however,the median of the degree-of-saliency map is calculated and used as thethreshold. The division process is executed by the partial-area divisionunit 402 in the same manner as in the first embodiment.

In the process executed by the pixel identification unit 403 in thefirst embodiment, the percentage of the number of extracted white pixelsis calculated for each partial area and is used as the distributioncharacteristic of the degrees of saliency. In the second embodiment, anabsolute value of a difference between the number of white pixels andthe number of black pixels in each partial area is used as thedistribution characteristic. The following is the reason why the processis executed in the embodiment.

Consider a case where the binarization is performed by using the medianof the degree-of-saliency map as the threshold as in the embodiment andwhere the number of pixels having the same degree of saliency as themedian is ignored. In this case, the white pixels and the black pixelshave the same number of pixels. To make the distribution characteristic(the absolute value of a difference between the number of white pixelsand the number of black pixels) equal in the partial areas in the casewhere the binary map is divided into a plurality of partial areas, thenumber of white pixels and the number of black pixels need to be equalto each other (or are distributed in percentages close to each other) inevery partial area. In other words, if the number of white pixels andthe number of black pixels are equal to each other (or are distributedin percentages close to each other) in every partial area, the spatialuniformity in distribution of the degrees of saliency is high.Contrarily, if the difference between the number of white pixels and thenumber of black pixels varies according to the partial areas of thebinary map, that is, in such a case where white pixels dominate in aspecific partial area and where black pixels dominate in another partialarea, the nonuniformity is high.

Specifically, in a case where the absolute value of the differencebetween the number of white pixels and the number of black pixels isused as the distribution characteristic, a value close to zero of thedistribution characteristic of a partial area indicates a high spatialuniformity in the distribution of the degrees of saliency, and a highvalue of the distribution characteristic of the partial area indicates ahigh spatial nonuniformity in the degrees of saliency. Thedistribution-characteristic extraction unit 104 binarizes thedegree-of-saliency map in this manner by using the median of thedegree-of-saliency map as the threshold and extracts, as thedistribution characteristic, the absolute value of the differencebetween the number of white pixels and the number of black pixels thatis calculated for each partial area. This enables thedegree-of-nonuniformity calculation unit 105 to calculate the degree ofthe spatial nonuniformity (or uniformity) at the subsequent stage.

Subsequently, the degree-of-nonuniformity calculation unit 105 adds upthe distribution characteristics (absolute values of the differenceseach between the number of white pixels and the number of black pixels)calculated for the respective partial areas. As described above, in thecase of the spatial uniformity in the distribution of the degrees ofsaliency, the distribution characteristic of each partial area takes ona value close to zero, and the value of the sum is thus a low value thatis close to zero. In the case of the spatial nonuniformity in thedistribution of the degrees of saliency, the distribution characteristicof each partial area takes on a high value, and the value of the sum isthus a high value. Accordingly, the degree-of-nonuniformity calculationunit 105 adds up the absolute values of the differences each between thenumber of white pixels and the number of black pixels obtained for therespective partial areas and outputs the value of the sum as the degreeof nonuniformity.

In computation in the embodiment as described above, the differencebetween the number of white pixels and the number of black pixels ineach partial area is calculated, and the difference values are added upto obtain the degree of nonuniformity. This leads to a smaller amount ofcomputation in the embodiment than in the first embodiment using thestandard deviation of the percentages of the white pixels. Inparticular, in a case where the main-subject detection apparatus of theembodiment functions integrally with an image capturing apparatus suchas a digital camera including the main-subject detection apparatusincorporated therein, the low load on the computation processing leadsto reduction of a period of time of detection processing and powersaving.

Third Embodiment

A third embodiment of the invention will be described. In the thirdembodiment, a plurality of degrees of nonuniformity are calculated. Inthis case, a division pattern used for dividing a degree-of-saliency mapinto partial areas is changed in accordance with the type of anapplication provided with a main-subject detection result, in otherwords, a plurality of division patterns are used to divide adegree-of-saliency map into partial areas. The components described inthe first and second embodiments are denoted by the same referencenumerals, and description thereof is omitted.

Examples of the application provided with a main-subject detectionresult include an application that adjusts AF on an area detected as amain-subject area and an application that performs the automatictracking on the main-subject area. It is conceivable that a featurerequired for the main subject detection varies with the application. Forexample, in the automatic tracking process, the uniform image and thescenic image are expected to be judged as absent-main-subject images asdescribed above based on the absence of an object to be tracked. Incontrast in the AF process, only the uniform image is expected to bejudged as the absent-main-subject image because the scenic image is nottroubled by an automatically focused horizon or skyline. As describedabove, the useful image judgment criterion varies with the applicationtype. In the embodiment, the pattern used for dividing adegree-of-saliency map into a plurality of partial areas is changed inaccordance with the application provided with a main-subject detectionresult.

FIG. 6 is a schematic block diagram illustrating the configuration of amain-subject detection apparatus 101 according to the third embodiment.The components described in the first embodiment are denoted by the samereference numerals, and description thereof is omitted. Hereinafter, adifference from the first embodiment will be described.

The main-subject detection apparatus 101 of the embodiment includes anapplication information acquisition unit 609 and a pattern selectionunit 610. The application information acquisition unit 609 acquiresinformation regarding the type of an application provided with amain-subject detection result and outputs the acquired information tothe pattern selection unit 610. Conceivable examples of the applicationprovided with a main-subject detection result include an AF application,an auto exposure (AE) application, and an automatic trackingapplication. The application information acquisition unit 609 hereinacquires, as the information regarding the application type, an imagecapturing mode set in a digital camera. For example, in a case where anAUTO mode is set as the image capturing mode, the application providedwith a main-subject detection result is the AF application. In theautomatic tracking mode, the automatic tracking process is provided witha main-subject detection result. Alternatively, a user may directlydesignate an application provided with a main-subject detection result.In this case, the user designation may be received, and information inthe designation may be acquired as application information.

The pattern selection unit 610 selects one of partial area patternsassociated with the application information acquired by the applicationinformation acquisition unit 609 and outputs the information aspartial-area-pattern selection information to thedistribution-characteristic extraction unit 104.

Application information is in advance associated with a correspondingone of the partial area patterns and information regarding therelationship there between is stored, for example, in a storage unit ofthe main-subject detection apparatus 101. Specifically, a partial areapattern suitable for an image to be judged as an absent-main-subjectimage has been determined in accordance with the type of the setapplication.

For example, an application requiring judgment of both of a uniformimage and a scenic image as absent-main-subject images is the automatictracking application, and an application requiring judgment of only auniform image as an absent-main-subject image is the AF application.When the automatic tracking is set as the application information, thepattern 902 illustrated in FIG. 9 is selected as the partial areapattern. In a case where the degree-of-saliency map is divided intopartial areas as in the pattern 902, the binary images of the respectiveuniform and scenic images each have a low degree of nonuniformitycalculated therefrom. The use of the feature thus enables the scorecalculation unit 106 to judge that each of the uniform image and thescenic image is likely to be an absent-main-subject image.

Likewise, when the AF is set as the application information, the pattern901 or the pattern 903 illustrated in FIG. 9 is selected as the partialarea pattern. For example, the degree-of-saliency map is divided intopartial areas as in the pattern 903, the binary image of only theuniform image has a low degree of nonuniformity calculated therefrom.The use of the feature thus enables the score calculation unit 106 tojudge that only the uniform image is likely to be an absent-main-subjectimage.

The partial-area pattern information selected by the pattern selectionunit 610 is output to the distribution-characteristic extraction unit104. The distribution-characteristic extraction unit 104 divides thedegree-of-saliency map into a plurality of partial areas in accordancewith the partial-area pattern information and extracts the distributioncharacteristic of the degrees of saliency in the same manner as in thefirst embodiment.

As described above, the embodiment enables judgment of anabsent-main-subject image performed in accordance with the type of theapplication provided with a main-subject detection result and thusenables highly flexible main subject detection.

Fourth Embodiment

To calculate the degrees of saliency in the description above, thehistograms are respectively generated for the data groups obtained fromthe two respective regions, an absolute difference between thehistograms is obtained for each bin, and the sum of the absolutedifferences is calculated. In the embodiment, however, thedegree-of-saliency calculation method is not limited thereto.

The form of the scan window used for calculating the degrees of saliencyis not limited to the form composed of the two regions of the first andsecond regions illustrated in FIG. 3. For example, a scan windowcomposed of one region may be used.

FIG. 7 is a diagram for explaining another example of the process foracquiring partial images. In this case, an input image 701 is scanned aplurality of times by using the scan window composed of a region 702that is a single region. An area corresponding to the region 702 in theinput image 701 is cut out at each pixel location in the input image 701to acquire a partial image, the region 702 corresponding to the firstregion. Pieces of data regarding the partial image are used as a firstdata group. The entire input image is set as a second region, and piecesof data regarding the entire input image are used as a second datagroup.

Each degree of saliency calculated by using the method illustrated inFIG. 3 is the degree of saliency calculated from a local point of viewbased on the first data group corresponding to the first region and thesecond data group corresponding to the second region neighboring thefirst region. In contrast, it can be said that the degree of saliencycalculated by using the method in FIG. 7 is the degree of saliencycalculated from a global point of view based on the difference betweenthe first region and the entire image because the same second data groupcorresponding to the entire image is used for the calculations. Thismethod reduces computation processing load because the data regardingthe entire image is always used as the second data group for thecalculations. Although the data regarding the entire image is hereinused as the second data group, a specific region that is fixed in theinput image may also be used as the second region.

In the description above, the nonuniformity of the distributioncharacteristics of the feature amounts is obtained for the entire inputimage, and whether the input image is a present-main-subject image or anabsent-main-subject image is distinguished. The embodiment, however, isapplicable to a configuration in which the nonuniformity of thedistribution characteristics of the feature amounts is obtained for notthe entire input image but a specific region of the input image to judgewhether a main subject is present in the region.

In the description above, the distribution characteristics in thedegree-of-saliency map generated from the input image are checked, but afeature amount other than the degree of saliency may be used to generatea map, and the distribution characteristics in the feature amount mapmay be checked. For example, the distribution characteristics in an edgeimage obtained by performing edge extraction may be checked. Since amain subject has an edge serving as an outline, it is expected that acertain number of high-intensity edges are extracted in apresent-main-subject image. In contrast, since a scenic image has edgesalong the horizon or a skyline, it is expected that the spatiallydistributed edges are extracted. The embodiment uses such a feature andis thus applicable to a configuration for judging whether an input imageis an absent-main-subject image. As described above, the embodiment iswidely applicable to a configuration for generating a map in which afeature amount at each pixel location in an input image is associatedwith the pixel location.

With the configurations described above, aspects of the presentinvention enable judgement of whether a main subject is present in aninput image.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While aspects of the present invention have been described withreference to exemplary embodiments, it is to be understood that theaspects of the invention are not limited to the disclosed exemplaryembodiments. The scope of the following claims is to be accorded thebroadest interpretation so as to encompass all such modifications andequivalent structures and functions.

What is claimed is:
 1. A main-subject detection method comprising:extracting feature amounts from pixels based on at least one region ofan input image and generating a map in which the extracted featureamounts respectively correspond to the pixels; dividing the generatedmap into a plurality of partial areas and extracting, on a per partialarea basis, distribution characteristics each based on at least acorresponding one of the feature amounts; calculating a degree ofnonuniformity of the extracted distribution characteristics of thefeature amounts, the degree of nonuniformity being a degree of spatialnonuniformity; and judging, based on the calculated degree ofnonuniformity, whether a main subject is present in the at least oneregion of the input image.